{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FtcmYWJc-FBb"
   },
   "source": [
    "# Practical 3A: Python packages and data \n",
    "\n",
    "Upon completion of this session you should be able to:\n",
    "\n",
    "   - use Python Packages to manipulate the data and files\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- Jupyter source file can be downloaded from clouddeakin SIT384 > weekly resources or https://github.com/gaoshangdeakin/SIT384-Jupyter\n",
    "- If you found any issue/bug for this document, please submit an issue at [https://github.com/gaoshangdeakin/SIT384/issues](https://github.com/gaoshangdeakin/SIT384/issues)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fb5TqlLE-FBo"
   },
   "source": [
    "## Content\n",
    "\n",
    "\n",
    "\n",
    "### Part 1 Python packages\n",
    "\n",
    "1.1 [Standard Libary](#standlib)\n",
    "\n",
    "1.2 [Third Party Packages](#3rdparty)\n",
    "\n",
    "1.3 [Importing a module](#importmod) \n",
    "\n",
    "\n",
    "### Part 2 Python Simple IO\n",
    "\n",
    "2.1 [Input](#input)\n",
    "\n",
    "2.2 [Output](#output)\n",
    "\n",
    "\n",
    "### Part 3 Datetime Module\n",
    "\n",
    "3.1 [Time](#time)\n",
    "\n",
    "3.2 [Date](#date)\n",
    "\n",
    "3.3 [Timedelta](#timedelta)\n",
    "\n",
    "3.4 [Formatting and Parsing](#parsing)\n",
    "\n",
    "### Part 4 Numpy Module\n",
    "\n",
    "4.1 [Importing Numpy](#importnp)\n",
    "\n",
    "4.2 [Numpy arrays](#nparray)\n",
    "\n",
    "4.3 [Manipulating arrays](#maninp)\n",
    "\n",
    "4.4 [Array Operations](#arrayop)\n",
    "\n",
    "4.5 [np.random](#random)\n",
    "\n",
    "\n",
    "### Part 5 Data Loading\n",
    "\n",
    "5.1 [TXT](#txt)\n",
    "\n",
    "5.2 [CSV](#csv)\n",
    "\n",
    "5.3 [JSON](#json)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nDEk8Gqz-FBx"
   },
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">1. Python packages</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ybgB1hro-FB2"
   },
   "source": [
    "After completing previous Python sessions, you should know about the syntax and semantics of the Python language. But apart from that, you should also learn about Python libraries and its packages to be able to code efficiently. Python’s standard library is very extensive, offering a wide range of facilities as indicated [here](https://docs.python.org/3.7/library/). The library contains built-in modules (written in C) that provide access to system functionality such as file I/O that would otherwise be inaccessible to Python programmers, as well as modules written in Python that provide standardized solutions for many problems that occur in everyday programming. Look at the [Python Standard Library Manual](https://docs.python.org/3.7/library/) to read more.\n",
    "\n",
    "In addition to the standard library, there is a growing collection of several thousand components (from individual programs and modules to packages and entire application development frameworks), available from the [Python Package Index](https://pypi.python.org/pypi)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ws56RKhy-FB7"
   },
   "source": [
    "<a id = \"standlib\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">1.1 Standard libraries</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YLU8yilr-FCa"
   },
   "source": [
    "For a complete list of Python standard library and their documentation look at the [Python Manual.](https://docs.python.org/3.7/library/) A few to mention are:\n",
    "\n",
    "* ``math`` for numeric and math-related functions and data types\n",
    "* ``urllib`` for fetching data across the web\n",
    "* ``datetime`` for manipulating dates and times\n",
    "* ``pickle`` and ``cPickle`` for serializing and deserializing data structures enabling us to save our variables on the disk and load them from the disk\n",
    "* ``os`` for os dependent functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D0gTVNG3-FCd"
   },
   "source": [
    "<a id = \"3rdparty\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">1.2 Third party packages</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G_FHaLGL-FCh"
   },
   "source": [
    "There are thousands of third party packages, each developed for a special task. Some of the useful libraries for data science are:\n",
    "\n",
    "* ``numpy`` is probably the most fundamental package for efficient scientific computing in Python\n",
    "* ``scipy`` is one of the core packages for scientific computation\n",
    "* ``pandas`` is a library for operating with table-like data structures called DataFrame object\n",
    "* ``matplotlib`` is a comprehensive plotting library\n",
    "* ``BeautifulSoup`` is an HTML and XML parser\n",
    "* ``scikit-learn`` is the most general machine learning library for Python\n",
    "* ``nltk`` is a toolkit for natural language processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n9-8E657-FCs"
   },
   "source": [
    "---\n",
    "<a id = \"importmod\"></a>\n",
    "### <span style=\"color:#0b486b\">1.3 Importing a module</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dDvdJbCz-FCv"
   },
   "source": [
    "To use a module, first you have to ``import`` it. There are different ways to import a module:\n",
    "\n",
    "* `import my_module`\n",
    "* `from my_module import my_function`\n",
    "* `from my_module import my_function as func`\n",
    "* `from my_module import submodule`\n",
    "* `from my_module import submodule as sub`\n",
    "* `from my_module import *`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-ZAPHJav-FCz"
   },
   "source": [
    "**`'import my_module'`** imports the module `'my_module'` and creates a reference to it in the namespace. For example `'import math'` imports the module `'math'` into the namespace. After importing the module this way, you can use the dot operator `(.)` to refer to the objects defined in the module. For example `'math.exp()'` refers to function `'exp()'` in module `'math'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f40P8yu7-FC3"
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "x = 2\n",
    "y1 = math.exp(x)\n",
    "y2 = math.log(x)\n",
    "\n",
    "print(\"e^{} is {} and log({}) is {}\".format(x, y1, x, y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "umlFNTcm-FDG"
   },
   "source": [
    "**`'from my_module import my_function'`** only imports the function `'my_function'` from the module `'my_module'` into the namespace. This way you won't have access to neither the module (since you have not imported the module), nor the other objects of the module. You can only have access to the object you have imported.\n",
    "\n",
    "You can use a comma to import multiple objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S68L0h56-FDK"
   },
   "outputs": [],
   "source": [
    "from math import exp\n",
    "\n",
    "x = 2\n",
    "y = exp(x)  # no need to math.exp()\n",
    "\n",
    "print(\"e^{} is {}\".format(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O7OwKRz0-FDV"
   },
   "source": [
    "**`'from my_module import my_function as func'`** imports the function `'my_function'` from module `'my_module'` but its identifier in the namespace is changed into `'func'`. This syntax is used to import submodules of a module as well. For example later you will see that nowadays it is almost a convention to import matplotlib.pyplot as plt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PkemZT1l-FDY"
   },
   "outputs": [],
   "source": [
    "# you can change the name of the imported object\n",
    "from math import exp as myfun\n",
    "\n",
    "x = 2\n",
    "y = myfun(x)\n",
    "\n",
    "print(\"e^{} is {}\".format(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yPV8jMzF-FDj"
   },
   "source": [
    "**`'from my_module import *'`** imports all the public objects defined in `'my_module'` into the namespace. Therefore after this statement you can simply use the plain name of the object to refer to it and there is no need to use the dot operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RRWQj8n9-FDq"
   },
   "outputs": [],
   "source": [
    "from math import *\n",
    "\n",
    "x = 2\n",
    "y1 = exp(x)\n",
    "y2 = log(x)\n",
    "\n",
    "print(\"e^{} is {} and log({}) is {}\".format(x, y1, x, y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y9wByq5c-FEM"
   },
   "source": [
    "\n",
    "## <span style=\"color:#0b486b\">2. Python simple input/output</span>\n",
    "\n",
    "Python uses input() and print() to do input and output, which have been introduced in previous two pracs. \n",
    "\n",
    "<a id = \"input\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.1 Input</span>\n",
    "\n",
    "`input()` asks the user for a string of data (ended with a newline), and simply returns the string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qXO1sN7o-FEw"
   },
   "source": [
    "<a id = \"output\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.2 output</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sb9kk_Uj-FE1"
   },
   "source": [
    "`print()` is the basic way to do output. To print multiple things on the same line separated by spaces, use commas between them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DC3XfzpT-FFB"
   },
   "source": [
    "Objects can be printed on the same line using the 'end' arguments. You can read the [print()](https://docs.python.org/3/library/functions.html#print) syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9KQpYWC7-FFF"
   },
   "outputs": [],
   "source": [
    "print('Sample is using the end=\\'\\,\\'')\n",
    "for i in range(10):\n",
    "    print(i, end=',')    \n",
    "print('\\nSample is using the end=\\' \\'')   \n",
    "for i in range(10):    \n",
    "    print(i, end=' ')\n",
    "print('\\nSample is without the end arguments') \n",
    "for i in range(10):    \n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U6o-Mudx-FFS"
   },
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">3. datetime module</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ro05oOUt-FFV"
   },
   "source": [
    "The datetime module includes functions and classes for date and time parsing, formatting, and arithmetic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gBxIzp5W-FFa"
   },
   "source": [
    "<a id = \"time\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.1 Time</span>\n",
    "\n",
    "Time values are represented with the time class. Times have attributes for hour, minute, second, and microsecond. They can also include time zone information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Up5dk2aY-FFd"
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "t = datetime.time(11, 21, 33)\n",
    "print(t)\n",
    "print('hour  :', t.hour)\n",
    "print('minute:', t.minute)\n",
    "print('second:', t.second)\n",
    "print('microsecond:', t.microsecond)\n",
    "print('tzinfo:', t.tzinfo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9mA0v-_s-FFq"
   },
   "source": [
    "<a id = \"date\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.2 Date</span>\n",
    "\n",
    "Calendar date values are represented with the date class. Instances have attributes for year, month, and day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hPdlr4qX-FFt"
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "today = datetime.date.today()\n",
    "print(today)\n",
    "print('ctime:', today.ctime())\n",
    "print('tuple:', today.timetuple())\n",
    "print('ordinal:', today.toordinal())\n",
    "print('Year:', today.year)\n",
    "print('Mon :', today.month)\n",
    "print('Day :', today.day)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PTbYXTCj-FF0"
   },
   "source": [
    "A way to create new date instances is using the `replace()` method of an existing date. For example, you can change the year, leaving the day and month alone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2bkkUju6-FF3"
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "d1 = datetime.date(2013, 3, 12)\n",
    "print('d1:', d1)\n",
    "\n",
    "d2 = d1.replace(year=2015)\n",
    "print('d2:', d2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ESNhrSou-FGJ"
   },
   "source": [
    "<a id = \"timedelta\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.3 timedelta</span>\n",
    "Using `replace()` is not the only way to calculate future/past dates. You can use datetime to perform basic arithmetic on date values via the timedelta class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fr7X-Pnq-FGL"
   },
   "outputs": [],
   "source": [
    "today = datetime.datetime.today()\n",
    "print(today)\n",
    "\n",
    "tomorrow = today + datetime.timedelta(days=1)  \n",
    "print(tomorrow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5wwW6zPU-FGn"
   },
   "source": [
    "<a id = \"parsing\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.4 Formatting and Parsing</span>\n",
    "\n",
    "The default string representation of a datetime object uses the ISO 8601 format (YYYY-MM-DDTHH:MM:SS.mmmmmm). Alternate formats can be generated using `strftime()`. Similarly, if your input data includes timestamp values parsable with `time.strptime()`, then `datetime.strptime()` is a convenient way to convert them to datetime instances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kRoDoulw-FGs"
   },
   "outputs": [],
   "source": [
    "today = datetime.datetime.today()\n",
    "print('ISO     :', today)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nLpPny1k-FG1"
   },
   "source": [
    "string from datetime object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0YNv0AR_-FG4"
   },
   "outputs": [],
   "source": [
    "str_format = \"%a %b %d %H:%M:%S %Y\"\n",
    "s = today.strftime(str_format)\n",
    "print('strftime:', s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dC2aAHmT-FHB"
   },
   "source": [
    "datetime object from string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qHTRTrSg-FHH"
   },
   "outputs": [],
   "source": [
    "print(s)\n",
    "\n",
    "d = datetime.datetime.strptime(s, str_format)\n",
    "print(d)\n",
    "print('strptime:', d.strftime(str_format))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UYPLivY6-FHO"
   },
   "outputs": [],
   "source": [
    "#Define a string variable \"s\", its value is 07/03/2017\n",
    "s = \"07/03/2017\"\n",
    "#Define the string format\n",
    "str_format = \"%m/%d/%Y\"\n",
    "\n",
    "\n",
    "d = datetime.datetime.strptime(s, str_format)\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fm6guAC1-FHe"
   },
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">4. Numpy module</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o1E02sGn-FHg"
   },
   "source": [
    "Python lists are very flexible for storing any sequence of Python objects. But usually flexibility comes at the price of performance and therefore Python lists are not ideal for numerical calculations where we are interested in performance. Here is where **NumPy** comes in. It adds support for large, multi-dimensional arrays and matrices, along with high-level mathematical functions to operate on these arrays to Python. \n",
    "\n",
    "Relying on `'BLAS'` and `'LAPACK'`, `'NumPy'` gives a functionality comparable with `'MATLAB'` to Python. NumPy facilitates advanced mathematical and other types of operations on large numbers of data. Typically, such operations are executed more efficiently and with less code than is possible using Python’s built-in sequences. It has become one of the fundamental packages used for numerical computations.\n",
    "\n",
    "In this tutorial we will review its basics, so to learn more about NumPy, visit [NumPy User Guide](http://docs.scipy.org/doc/numpy/user/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Hc_bsO3l-FHj"
   },
   "source": [
    "<a id = \"importnp\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">4.1 Importing Numpy</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yWUvtzi8-FHm"
   },
   "source": [
    "First we have to import a package to be able to use it. NumPy is imported with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2-B8prOg-FHq"
   },
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a39r4M3p-FH1"
   },
   "source": [
    "It is the convention to import it like with an alias:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SKj133sh-FH7"
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uAmsO-V8-FIR"
   },
   "source": [
    "<a id = \"nparray\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">4.2 Numpy arrays</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tlnX1eG8-FIT"
   },
   "source": [
    "The core of NumPy is its arrays. You can create an array from a Python list or tuple using `'array'` function. They work similarly to lists apart from the fact that:\n",
    "\n",
    "* you can easily perform element-wise operation on them, and\n",
    "* unlike lists, they should be pre-allocated.\n",
    "\n",
    "The first point is further explained in the following array operations section. The second point means that you there is no equivalent to list append for arrays. The size of the arrays is known at the time it is defined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ATvr-1ZS-FIW"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.2.1 create an array from a list</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xC6z9Xyo-FIY"
   },
   "outputs": [],
   "source": [
    "x = [1, 7, 3, 4, 0, -5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gdsnKRWa-FIi"
   },
   "outputs": [],
   "source": [
    "y = np.array(x)\n",
    "type(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nnBy-g5b-FIq"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.2.2 create an array using a range</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OBt00bhq-FIs"
   },
   "outputs": [],
   "source": [
    "range(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e1KYDBLq-FI7"
   },
   "outputs": [],
   "source": [
    "print(np.array(range(5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LhL0M03i-FJD"
   },
   "outputs": [],
   "source": [
    "print(np.arange(2, 3, 0.2))  #Why is there no value 3.0 in the output?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_SQTHmfL-FJL"
   },
   "outputs": [],
   "source": [
    "print(np.linspace(2, 3, 5))    # returns numbers spaced evenly on a linear scale, both endspoints are included\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JDpURgHfM8-D"
   },
   "source": [
    "Just try to change the variable value 5 with 1, 2, 4  or 10? What pattern could you find? Could you guess what is the function of **linspace**, if without the given comments? \n",
    "Then, you can try to use the same method to learn what is the function of **logspace**?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ol089FmC-FJS"
   },
   "outputs": [],
   "source": [
    "print(np.logspace(2, 3, 5))   # returns numbers spaced evenly on a log scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hXqpHuub-FJZ"
   },
   "source": [
    "**Note:** If you need any help on how to use a function or what it does, you can IPython help. Just add a question mark (?) at the end of the function and execute the cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xFYTD-O--FJb"
   },
   "outputs": [],
   "source": [
    "np.logspace?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PnwvGJX4-FJj"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.2.3 create a prefilled array</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zSvnqTuR-FJn"
   },
   "outputs": [],
   "source": [
    "print(np.zeros(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MuWwpVx2-FJ0"
   },
   "outputs": [],
   "source": [
    "print(\"The 1st sample is\",np.ones(5, dtype=int))   # you can specify the data type, default is float\n",
    "print(\"The 2nd sample is \",np.ones(5, dtype=float)) \n",
    "print(\"The 3rd sample is \",np.ones((5,5), dtype=int)) \n",
    "print(\"The 4th sample is \",np.ones((5,5,5), dtype=int)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1crrWNwwO0BP"
   },
   "outputs": [],
   "source": [
    "np.ones?\n",
    "#You can use this command to learn what is the function of np.ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g7FgDgFa-FJ7"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.2.4 `'mgrid'`</span>\n",
    "similar to meshgrid in MATLAB:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U5OdZS-L-FKB"
   },
   "outputs": [],
   "source": [
    "x, y = np.mgrid[0:5, 0:3]\n",
    "\n",
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vI0XK1HEPpRb"
   },
   "outputs": [],
   "source": [
    "np.mgrid?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following picture might give a better example of how meshgrid works. Details are available [here]( \n",
    "https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html).\n",
    "\n",
    "<img src=\"./images/p03/meshgrid.jpg\" width=\"60%\" height=\"60%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BK24IR2b-FKJ"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.2.5 array attributes</span>\n",
    "\n",
    "NumPy arrays have multiple attributes and methods. The cell below shows a few of them. You can press tab after typing the dot operator `'(.)'` to use IPython auto-complete and see the rest of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MirmVfAm-FKN"
   },
   "outputs": [],
   "source": [
    "y = np.array([3, 0, -4, 6, 12, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lPHoezhC-FKW"
   },
   "outputs": [],
   "source": [
    "print(\"number of dimensions:\\t\", y.ndim)        \n",
    "print(\"dimension of the array:\", y.shape)       \n",
    "print(\"numerical data type:\\t\", y.dtype)\n",
    "print(\"maximum of the array:\\t\", y.max())       \n",
    "print(\"index of the array max:\", y.argmax())    \n",
    "print(\"mean of the array:\\t\", y.mean())      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CrbiDDhr-FKk"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.2.6 Multi-dimensional arrays</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qDBhj18y-FKn"
   },
   "source": [
    "You can define arrays with 2 (or higher) dimensions in numpy:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ltKaGI6K-FKo"
   },
   "source": [
    "##### from lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d8M2wkbR-FKr"
   },
   "outputs": [],
   "source": [
    "x = [[1, 2, 10, 20], [3, 4, 30, 40]]\n",
    "y = np.array(x)\n",
    "print(y)\n",
    "print()\n",
    "print(y.ndim, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XmXmLGk1-FKz"
   },
   "source": [
    "##### pre-filled "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qiG5NGcB-FK2"
   },
   "outputs": [],
   "source": [
    "x = np.ones((3, 5), dtype='int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qSRQFimN-FK8",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(x)\n",
    "print()\n",
    "print(x.ndim, x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "foS348qb-FLB"
   },
   "source": [
    "##### `'diag()'`\n",
    "diagonal matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M-ttVrqJ-FLN"
   },
   "outputs": [],
   "source": [
    "np.diag([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3fRD43wu-FLY"
   },
   "source": [
    "<a id = \"maninp\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">4.3 Manipulating arrays</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cgGZKWuW-FLa"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.1 Indexing</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "reg5bJIY-FLc"
   },
   "source": [
    "Similar to lists, you can index elements in an array using `'[]'` and indices:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "47U6VVq--FLf"
   },
   "source": [
    "If `'x'` is a 1-dimensional array, `'x[i]'` will index `'ith'` element of `'x'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WI3XEpO0-FLi"
   },
   "outputs": [],
   "source": [
    "x = np.array([2, 8, -2, 4, 3])\n",
    "print(x[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kOhiutVu-FLp"
   },
   "source": [
    "If 'x' is a 2-dimensional arrray:\n",
    "\n",
    "* '`x[i, j]'` or `'x[i][j]'` will index the element in `'ith'` row and `'jth'` column\n",
    "* '`x[i, :]'` will index the `'ith'` row \n",
    "* `'x[:, j]'` will index `'jth'` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5dZ8bMdA-FLr"
   },
   "outputs": [],
   "source": [
    "x = np.array([[7, 6, 8, 6, 4],\n",
    "              [4, 7, -2, 0, 9]])\n",
    "              \n",
    "print(x[1, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RYHDHduJ-FLw"
   },
   "outputs": [],
   "source": [
    "print(x[1, :])      # or x[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WUyelaZV-FL1"
   },
   "outputs": [],
   "source": [
    "print(x[:, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YMP2cU6y-FMB"
   },
   "source": [
    "Arrays can also be indexed with other arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QgEan722-FMC"
   },
   "outputs": [],
   "source": [
    "x = np.array([2, 8, -2, 4, 3, 9, 0])\n",
    "\n",
    "idx1 = [1, 3, 4]        # list\n",
    "idx2 = np.array(idx1)   # array\n",
    "\n",
    "print(x[idx1], x[idx2])\n",
    "x[idx2] = 0\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0KBUG3Eg-FMI"
   },
   "source": [
    "You can also index masks. The index mask should be a NumPy arrays of data type Bool. Then the element of the array is selected only if the index mask at the position of the element is True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X1I8PH1Q-FMK"
   },
   "outputs": [],
   "source": [
    "x = np.array([2, 8, -2, 4, 3, 9, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7ge96tsf-FMQ"
   },
   "outputs": [],
   "source": [
    "mask = np.array([False, True, True, False, False, True, False])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PEoatrjz-FMU"
   },
   "outputs": [],
   "source": [
    "x[mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rHObUufk-FMb"
   },
   "source": [
    "Combining index masks with comparison operaors enabels you to conditionally slecect elements of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ylAy1AHp-FMd"
   },
   "outputs": [],
   "source": [
    "x = np.array([2, 8, -2, 4, 3, 9, 0])\n",
    "mask = (x>=2) * (x<9)\n",
    "x[mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HA-J3zZr-FMq"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.2 Slicing</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "blX9Xzbl-FMs"
   },
   "source": [
    "Similar to Python lists, arrays can also be sliced:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QJ550Wak-FMu"
   },
   "outputs": [],
   "source": [
    "x = np.array([2, 8, -2, 4, 3, 9, 0])\n",
    "\n",
    "print(x[3:])    # slicing\n",
    "print(x[3:7:2])  # slicing with a specified step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w8V9DnNs-FMz"
   },
   "outputs": [],
   "source": [
    "x = np.array([[7, 6, 8, 6, 4, 3],\n",
    "              [4, 7, 0, 5, 9, 5],\n",
    "              [7, 3, 6, 3, 5, 1]])\n",
    "              \n",
    "\n",
    "print(x[1, 1:4])\n",
    "print()\n",
    "print(x[:2, 1::2])    # rows zero up to 2, cols 1 up to end with a step=2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hWTmUPd6-FM5"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.3 Iteration over items</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Eu9gjTKo-FM7"
   },
   "source": [
    "Since most of NumPy functions are capable of operating on arrays, in many cases iteration over items of an arrays can be (and should be) avoided. Otherwise it is pretty much similar to iterating over values of a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eayPM8on-FNA"
   },
   "outputs": [],
   "source": [
    "a = np.arange(0, 50, 7)\n",
    "print(a)\n",
    "for item in a:\n",
    "    print(item,) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uEEZs2Pd-FNG"
   },
   "source": [
    "Of course you could iterate over items using their indices too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W35khYsr-FNJ"
   },
   "outputs": [],
   "source": [
    "a = np.arange(0, 50, 7)\n",
    "for i in range(a.shape[0]):\n",
    "    print(a[i],)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E9efgG2f-FNQ"
   },
   "source": [
    "There are also many functions for manipulating arrays. The most used ones are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jq2e_fTx-FNS"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.4 `copy()`</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Yqcj120w-FNU"
   },
   "source": [
    "**Remember** that assignment operator is not an equivalence for copying arrays. In fact Python does not pass the values. It passess the references as array is mutable class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Wkx8l22X-FNa"
   },
   "outputs": [],
   "source": [
    "x = [1, 2, 3]\n",
    "y = x\n",
    "print(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "69hYgW58-FNf"
   },
   "outputs": [],
   "source": [
    "y[0] = 0       # now we alter an element of y\n",
    "print(x, y)     # note that x has changed as well"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Oof00tVc-FNq"
   },
   "source": [
    "Same is true for numpy arrays. That's why if you need a copy of an array, you should use `'copy()'` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ir5585gc-FNs"
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "y = x\n",
    "\n",
    "y[0] = 0       # now we alter an element of y\n",
    "print(x, y)     # note that x has changed as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Uo-iZzvC-FN3"
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "y = x.copy()  # or np.copy(x)\n",
    "y[0] = 0\n",
    "\n",
    "print(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tSrPOa8t-FN_"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.5 `reshape()`</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8tnEtiPp-FOC"
   },
   "outputs": [],
   "source": [
    "x1 = np.arange(6)\n",
    "x2 = x1.reshape((2, 3))    # or np.reshape(x1, (2, 3))\n",
    "\n",
    "print(x1)\n",
    "print()\n",
    "print(x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7_aHU31V-FOL"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.6 `astype()`</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "76fjepzM-FON"
   },
   "source": [
    "Used for type casting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k1GoBiQV-FOO"
   },
   "outputs": [],
   "source": [
    "x1 = np.arange(5)\n",
    "x2 = x1.astype(float)\n",
    "\n",
    "print(type(x1), x1)\n",
    "print(type(x2), x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k6bOoR14-FOU"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.3.7 `T`</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oe1PUocl-FOc"
   },
   "source": [
    "transpose method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DXLlgFaT-FOe"
   },
   "outputs": [],
   "source": [
    "x1 = np.random.randint(5, size=(2, 4))\n",
    "x2 = x1.T\n",
    "\n",
    "print(x1)\n",
    "print()\n",
    "print(x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "axdyp7nJ-FOi"
   },
   "source": [
    "<a id = \"arrayop\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">4.4 Array operations</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pdnp48Ov-FOp"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.4.1 Arithmetic operators</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tbF09PSC-FOr"
   },
   "source": [
    "Arrays can be added, subtracted, multiplied and divided using +, -, \\* and, /. Operations done by these operators are **element wise**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Rrdn8VQE-FOt"
   },
   "outputs": [],
   "source": [
    "x1 = np.array([[2, 3, 5, 7], \n",
    "               [2, 4, 6, 8]], dtype=float)\n",
    "x2 = np.array([[6, 5, 4, 3], \n",
    "               [9, 7, 5, 3]], dtype=float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NsdLlPTm-FO1"
   },
   "outputs": [],
   "source": [
    "print(x1)\n",
    "print()\n",
    "print(x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6NWyxuog-FO7"
   },
   "outputs": [],
   "source": [
    "print(x1 + x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xb4lLmlt-FPG"
   },
   "outputs": [],
   "source": [
    "print(x1 - x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7NM1P_u6-FPM"
   },
   "outputs": [],
   "source": [
    "print(x1 * x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pBirpVSg-FPR"
   },
   "outputs": [],
   "source": [
    "print(x1 / x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BWYmg3gc-FPZ"
   },
   "outputs": [],
   "source": [
    "print(3 + x1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bHFn1Mxt-FPd"
   },
   "outputs": [],
   "source": [
    "print(3 * x1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Piegik6m-FPl"
   },
   "outputs": [],
   "source": [
    "print(3 / x1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2xqoJwUZ-FPr"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">4.4.2 Boolean operators</span>\n",
    "\n",
    "Much like arithmetic operators discussed above, boolean (comparison) operators perform element-wise on arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aM5g731W-FPs"
   },
   "outputs": [],
   "source": [
    "x1 = np.array([2, 3, 5, 7])\n",
    "x2 = np.array([2, 4, 6, 7])\n",
    "y = x1<x2\n",
    "\n",
    "print( y, y.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cnIFJb0j-FPw"
   },
   "source": [
    "use methods `'.any()'` and `'.all()'` to return a single boolean value indicating whether any or all values in the array are True respectively. This value in turn can be used as a condition for an `'if'` statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7NGdUs29-FPy"
   },
   "outputs": [],
   "source": [
    "print (y.all())\n",
    "print (y.any())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xy2gjc2T-FP3"
   },
   "source": [
    "NumPy has many other functions that you can read about them in [NumPy User Guide](http://docs.scipy.org/doc/numpy/user/). Specially read about:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0lc_o_C5-FP4"
   },
   "source": [
    "* `np.unique`, returns unique elements of an array\n",
    "* `np.flatten`, flattens a multi-dimensional array\n",
    "* `np.mean`, `np.std`, `np.median`\n",
    "* `np.min`, `np.max`, `np.argmin`, `np.argmax`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xTcRRTbn-FP5"
   },
   "source": [
    "<a id = \"random\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">4.5 np.random</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MPPLTWcv-FP7"
   },
   "source": [
    "NumPy has a module called `random` to generate arrays of random numbers. There are different ways to generate a random number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OlyjQz8z-FP8"
   },
   "outputs": [],
   "source": [
    "print( np.random.rand())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IxnFBbRv-FQD"
   },
   "outputs": [],
   "source": [
    "# 2x5 random array drawn from standard normal distribution\n",
    "print( np.random.random([2, 5]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YZ_OPkSs-FQI"
   },
   "outputs": [],
   "source": [
    "# 2x5 random array drawn from standard normal distribution\n",
    "print (np.random.rand(2, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the difference between np.random.rand() and np.random.random()?\n",
    "\n",
    "Both functions generate samples from the uniform distribution on \\[0, 1). The only difference is in how the arguments are handled. With numpy.random.rand, the length of each dimension of the output array is a separate argument. With numpy.random.random_sample or numpy.random.random_sample (numpy.random.random is actually an alias for numpy.random.random_sample), the shape argument is a single tuple. \n",
    "\n",
    "(source: [stackoverflow](https://stackoverflow.com/questions/47231852/np-random-rand-vs-np-random-random))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9GIPOTP_-FQM"
   },
   "outputs": [],
   "source": [
    "# 2x5 random array drawn from a uniform distribution on {0, 1, 2, ..., 9}\n",
    "print (np.random.randint(10, size=[2, 5])) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C9UFe6xH-FQQ"
   },
   "source": [
    "##### <span style=\"color:#0b486b\">4.5.1 Random seed</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yqKfnar6-FQR"
   },
   "source": [
    "Random numbers generated by computers are not really random. They are called pseudo-random. Thus we can set the random generator to generate the same set of random numbers every time. This is useful while testing the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d55atcWW-FQS"
   },
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    print (np.random.random(),)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IEhNqhH2-FQX"
   },
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    np.random.seed(100)\n",
    "    print (np.random.random(),)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "05sezgN2-FQ9"
   },
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">5. File I/O</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z6iEGtaO-FQ-"
   },
   "source": [
    "For Online platforms such as Google Colab or IBM Cloud, it is important for you to get familiar with the provided data storage or cloud data storage function. Alternatively, you can directly save the file and load into your Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DopvIhrG-FRA",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install wget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QXFY49LC-FRE"
   },
   "source": [
    "Then you can download the file into the online platform's file system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ab6v2GxS-FRF"
   },
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://raw.githubusercontent.com/gaoshangdeakin/SIT384/master/csv_data1.csv'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uluOknMC-FRJ"
   },
   "source": [
    "<a id = \"txt\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">5.1 TXT</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o1ZysbwZ-FRK"
   },
   "source": [
    "TXT file format is the most simplestic way to store data. \n",
    "\n",
    "Load a TXT file with `'np.loadtxt()'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q2zCykcT-FRL"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# This code is for local PC\n",
    "# x = np.loadtxt(\"data/txt_data1.txt\")\n",
    "\n",
    "# The following code for online platform such as colab or IBM Cloud\n",
    "link_to_data = 'https://raw.githubusercontent.com/gaoshangdeakin/SIT384/master/txt_data1.txt'\n",
    "DataSet = wget.download(link_to_data)\n",
    "\n",
    "x = np.loadtxt(\"txt_data1.txt\")\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iZZfyuL1-FRP"
   },
   "source": [
    "Save a TXT file with `'np.savetxt()'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-RkfeDje-FRR"
   },
   "outputs": [],
   "source": [
    "y = np.random.randint(10, size=5)\n",
    "np.savetxt(\"txt_data2.txt\", y)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "OT2pfvIA-FRW"
   },
   "source": [
    "<a id = \"csv\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">5.2 CSV</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XN8xU7cR-FRX"
   },
   "source": [
    "Comma Separated Values format and its variations are one the most used file format to store data.\n",
    "\n",
    "You can use `'np.genfromtxt()'` to read a CSV file:\n",
    "\n",
    "**NOTE:** The best way to read CSV and XLS files is using **pandas** package that will be introduced later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gHHWnj8r-FRZ"
   },
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://raw.githubusercontent.com/gaoshangdeakin/SIT384/master/csv_data1.csv'\n",
    "DataSet = wget.download(link_to_data)\n",
    "\n",
    "print(DataSet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PlB0GcDT-FRd"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "x = np.genfromtxt(\"csv_data1.csv\", delimiter=\",\")\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eMR_478x-FRk"
   },
   "source": [
    "Use `'np.savetxt()'` to save a 2d-array in a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BvkGTJF_-FRl"
   },
   "outputs": [],
   "source": [
    "x = np.random.randint(10, size=(6,4))\n",
    "np.savetxt(\"csv_data2.csv\", x, delimiter=',')\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jWORjUcv-FRp"
   },
   "source": [
    "<a id = \"json\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">5.3 JSON</span>\n",
    "\n",
    "\n",
    "JSON is the most used file format when dealing with web services. \n",
    "\n",
    "To read a JSON file, use `'json'` package and `'load()'` function, or `'loads()'` if the data is serialized. It reads the data and parses it into a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yABGeta3-FRq"
   },
   "outputs": [],
   "source": [
    "link_to_data = 'https://raw.githubusercontent.com/gaoshangdeakin/SIT384/master/json_data1.json'\n",
    "DataSet = wget.download(link_to_data)\n",
    "\n",
    "print(DataSet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7FAA_6uT-FRu"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "with open(\"json_data1.json\", 'rb') as fp:\n",
    "    fcontent = fp.read()\n",
    "# data = json.loads(fcontent)\n",
    "data = json.loads(fcontent.decode('utf-8'))\n",
    "data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DWTeTfFq-FRz"
   },
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hc-aODdh-FR2"
   },
   "outputs": [],
   "source": [
    "data['phoneNumbers']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dOuOtK50-FR5"
   },
   "source": [
    "You can also write a python dictionary into a JSON file. To do this use `'dump()'` or `'dumps()'` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GuglShkt-FR6"
   },
   "outputs": [],
   "source": [
    "data = [{'Name': 'Zara', 'Age': 7, 'Class': 'First'}, \n",
    "        {'Name': 'Lily', 'Age': 9, 'Class': 'Third'}];\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ql8yHsKw-FR9"
   },
   "outputs": [],
   "source": [
    "with open(\"json_data_now.json\", 'w') as fp:\n",
    "    json.dump(data, fp)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "SIT742P02A-PackagesAndData.ipynb",
   "provenance": [
    {
     "file_id": "https://github.com/tulip-lab/sit742/blob/master/Jupyter/SIT742P02A-PackagesAndData.ipynb",
     "timestamp": 1552262266669
    },
    {
     "file_id": "https://github.com/tulip-lab/sit742/blob/master/Jupyter/SIT742P02A-PackagesAndData.ipynb",
     "timestamp": 1551757765883
    }
   ],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
